Eye tracking may be used in computing systems to detect user inputs made by gaze interactions with displayed user interface objects. Eye tracking systems may utilize images of a user's pupil and corneal reflections from glint light sources to determine a line of sight of the user's eye. However, different users may have anatomical differences that can affect such a determination. For example, the location of the fovea on the retina may differ from person to person. As such, the visual axis of the eye (i.e. the axis extending from the physical object imaged on the user's fovea to the fovea) may be offset from the optical axis of the eye (i.e. the axis extending through the center of the pupil to the back pole of the eyeball). As the image data is used to determine an estimate of the optical axis of the eye, such an offset may complicate the accurate estimate of an actual location at which the user is gazing.